{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Learning for NLP with PyTorch\n",
    "\n",
    "## Introduction to PyTorch\n",
    "\n",
    "省略\n",
    "\n",
    "## Deep Learning with PyTorch\n",
    "\n",
    "### Deep Learning Building Blocks: Affine maps, non-linearities and objectives\n",
    "\n",
    "Deep learning consists of composing linearities with non-linearities in clever ways. Non-linearities allows for powerful models.\n",
    "\n",
    "#### Affine Maps\n",
    "\n",
    "Affine map is a function *f(x)* where *f(x)=Ax+b* for a matrix A and vectors x, b.\n",
    "\n",
    "PyTorch and most other deep learning frameworks do things a little differently than traditional linear algebra. It **maps the rows of the input instead of the columns**. That is, the i'th row of the output below is the mapping of the i'th row of input under A, plus the bias term."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x112849210>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "torch.manual_seed(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 0.1755, -0.3268, -0.5069],\n",
      "        [-0.6602,  0.2260,  0.1089]], grad_fn=<AddmmBackward>)\n"
     ]
    }
   ],
   "source": [
    "lin = nn.Linear(5, 3)\n",
    "data = torch.randn(2, 5)\n",
    "print(lin(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Non-Linearities\n",
    "\n",
    "We know that composing affine maps gives you an affine map. So if our model is a long chain of affine compositions, this adds no new power to your model than just doing a single affine map.\n",
    "\n",
    "If we introduce non-linearitiesin between the affine layers, we can build much more powerful models.\n",
    "\n",
    "There are a few core non-linearities: **tanh(x), sigmoid(x), ReLU(x)**, etc. They are the most common since they have gradients that are easy to compute, and computing gradients is essential for learning.\n",
    "\n",
    "In PyTorch, most non-linearities are in **torch.functional**, and they don't have parameters (weights) that need to be updated during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.5404, -2.2102],\n",
      "        [ 2.1130, -0.0040]])\n",
      "tensor([[0.0000, 0.0000],\n",
      "        [2.1130, 0.0000]])\n"
     ]
    }
   ],
   "source": [
    "data = torch.randn(2, 2)\n",
    "print(data)\n",
    "print(F.relu(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Softmax and Probabilities\n",
    "\n",
    "The function Softmax(x) is also just a non-linearity, but it is special in that it usually is the last operation done in a network. This is because it takes in a vector of real numbers and returns a probability distribution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 刘尧：注意！！！log_softmax就是在softmax外进行个log操作，不过是同时进行的，而非分别进行。因为对于分类来说，一般使用对数似然激活函数，就一次性都计算了。\n",
    "\n",
    "> 注意！ PyTorch里提供了**torch.nn.CrossEntropyLoss(This criterion combines nn.LogSoftmax() and nn.NLLLoss() in one single class)**，其整合了上面的步骤。这和tf.nn.softmax_cross_entropy_with_logits的功能是一致的。\n",
    "\n",
    "> 必须明确一点：在PyTorch中若模型使用CrossEntropyLoss这个loss函数，则**不应该在最后一层再使用softmax进行激活**。CrossEntropyLoss() is the same as NLLLoss(), except it does the log softmax for you.\n",
    "\n",
    "> 而在Keras中，我们通常在最后一层使用softmax进行激活，保证输出神经元的值即分类的概率值，然后在compile中使用损失函数categorical_crossentropy，这符合常理。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([ 1.3800, -1.3505,  0.3455,  0.5046,  1.8213])\n",
      "tensor([0.2948, 0.0192, 0.1048, 0.1228, 0.4584])\n",
      "tensor(1.)\n",
      "tensor([-1.2214, -3.9519, -2.2560, -2.0969, -0.7801])\n"
     ]
    }
   ],
   "source": [
    "data = torch.randn(5)\n",
    "print(data)\n",
    "print(F.softmax(data, dim=0))\n",
    "print(F.softmax(data, dim=0).sum())\n",
    "print(F.log_softmax(data, dim=0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Objective Functions\n",
    "\n",
    "省略\n",
    "\n",
    "### Optimization and Training\n",
    "\n",
    "省略\n",
    "\n",
    "### Creating Network Components in PyTorch\n",
    "\n",
    "#### Example: Logistic Regression Bag-of-Words classifier\n",
    "\n",
    "Our model will map a sparse BoW representation to log probabilities over labels. Say our entire vocab is two words \"hello\" and \"world\", with indices 0 and 1 respectively. In general, the BoW vector for sentence is **[Count(hello), Count(world)]**. \n",
    "\n",
    "Denote this BoW vector as x. The output of our network is **logSoftmax(Ax+b)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [('me gusta comer en la cafeteria'.split(), 'SPANISH'),\n",
    "        ('Give it to me'.split(), 'ENGLISH'),\n",
    "        ('No creo que sea una buena idea'.split(), 'SPANISH'),\n",
    "        ('No it is not a good idea to get lost at sea'.split(), 'ENGLISH')]\n",
    "test_data = [('Yo creo que si'.split(), 'SPANISH'),\n",
    "             ('it is lost on me'.split(), 'ENGLISH')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'me': 0, 'gusta': 1, 'comer': 2, 'en': 3, 'la': 4, 'cafeteria': 5, 'Give': 6, 'it': 7, 'to': 8, 'No': 9, 'creo': 10, 'que': 11, 'sea': 12, 'una': 13, 'buena': 14, 'idea': 15, 'is': 16, 'not': 17, 'a': 18, 'good': 19, 'get': 20, 'lost': 21, 'at': 22, 'Yo': 23, 'si': 24, 'on': 25}\n"
     ]
    }
   ],
   "source": [
    "word2index = {}\n",
    "for sent, _ in data + test_data:  # vocab来自于所有数据：训练数据和测试数据\n",
    "    for word in sent:\n",
    "        if word not in word2index:\n",
    "            word2index[word] = len(word2index)  # 巧妙！\n",
    "print(word2index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "VOCAB_SIZE = len(word2index)\n",
    "NUM_LABEL = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BoWClassifier(nn.Module):\n",
    "    \n",
    "    def __init__(self, num_labels, vocab_size):\n",
    "        super(BoWClassifier, self).__init__()\n",
    "        self.linear = nn.Linear(vocab_size, num_labels)\n",
    "        # Note: the non-linearity log softmax does not have parameters! So we don't need to worry about them here\n",
    "        \n",
    "    def forward(self, bow_vec):\n",
    "        return F.log_softmax(self.linear(bow_vec), dim=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_bow_vector(sentence, word2index):\n",
    "    vec = torch.zeros(len(word2index))\n",
    "    for word in sentence:\n",
    "        vec[word2index[word]] += 1\n",
    "    return vec.view(1, -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "label2index = {\"SPANISH\": 0, \"ENGLISH\": 1}\n",
    "def make_target(label, label2index):\n",
    "    return torch.LongTensor([label2index[label]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parameter containing:\n",
      "tensor([[ 0.1194,  0.0609, -0.1268,  0.1274,  0.1191,  0.1739, -0.1099, -0.0323,\n",
      "         -0.0038,  0.0286, -0.1488, -0.1392,  0.1067, -0.0460,  0.0958,  0.0112,\n",
      "          0.0644,  0.0431,  0.0713,  0.0972, -0.1816,  0.0987, -0.1379, -0.1480,\n",
      "          0.0119, -0.0334],\n",
      "        [ 0.1152, -0.1136, -0.1743,  0.1427, -0.0291,  0.1103,  0.0630, -0.1471,\n",
      "          0.0394,  0.0471, -0.1313, -0.0931,  0.0669,  0.0351, -0.0834, -0.0594,\n",
      "          0.1796, -0.0363,  0.1106,  0.0849, -0.1268, -0.1668,  0.1882,  0.0102,\n",
      "          0.1344,  0.0406]], requires_grad=True)\n",
      "Parameter containing:\n",
      "tensor([0.0631, 0.1465], requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "model = BoWClassifier(NUM_LABEL, VOCAB_SIZE)\n",
    "for param in model.parameters():\n",
    "    print(param)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.5378, -0.8771]])\n"
     ]
    }
   ],
   "source": [
    "with torch.no_grad():\n",
    "    sample = data[0]\n",
    "    bow_vector = make_bow_vector(sample[0], word2index)\n",
    "    log_probs = model(bow_vector)\n",
    "    print(log_probs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_function = nn.NLLLoss()\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.1)\n",
    "for epoch in range(100):\n",
    "    for instance, label in data:\n",
    "        # Step 1. Remember that PyTorch accumulates gradients. We need to clear them out before each instance.\n",
    "        model.zero_grad()\n",
    "        # Step 2.\n",
    "        bow_vec = make_bow_vector(instance, word2index)\n",
    "        target = make_target(label, label2index)\n",
    "        # Step 3. Run forward pass\n",
    "        log_probs = model(bow_vec)\n",
    "        # Step 4. Compute the loss, gradients, and update the parameters\n",
    "        loss = loss_function(log_probs, target)\n",
    "        loss.backward()\n",
    "        optimizer.step()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.2093, -1.6669]])\n",
      "tensor([[-2.5330, -0.0828]])\n"
     ]
    }
   ],
   "source": [
    "with torch.no_grad():\n",
    "    for instance, label in test_data:\n",
    "        bow_vec = make_bow_vector(instance, word2index)\n",
    "        log_probs = model(bow_vec)\n",
    "        print(log_probs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Word Embeddings: Encoding Lexical Semantics\n",
    "\n",
    "A fundamental linguistic assumption: words appearing in similar contexts are related to each other semantically. This is called the **distributional hypothesis**.\n",
    "\n",
    "### Getting Dense Word Embeddings\n",
    "\n",
    "Suppose we have seen the sentences in training data:\n",
    "\n",
    "* The mathematician ran to the store.*\n",
    "\n",
    "* The physicist ran to the store.*\n",
    "\n",
    "* The mathematician solved the open problem.*\n",
    "\n",
    "Now suppose we get a new sentence never seen before in training data:\n",
    "\n",
    "* The physicist solved the open problem.*\n",
    "\n",
    "How could we actually encode semantic similarity in words? Maybe we think up some semantic attributions. For example, in the sentences above, **we can give 'mathematician' and 'physicist' a high score for the 'is able to run' semantic attribute**. Think of some other attributes, and imagine what you might score some common words on those attributes.\n",
    "\n",
    "If each attribute is a dimension, then we might give each word a vector, like this:\n",
    "\n",
    "![](./image/word_embedding_attribute.png)\n",
    "\n",
    "The attributes are: **'can run', 'likes coffee', 'majored in Physics'**, ..., which are dimensions.\n",
    "\n",
    "A big pain: we could think of thousands of different semantic attributes that might be relevant to determining similarity.\n",
    "\n",
    "Idea of deep learning: the neural network learns representations of the features, rather than requiring the programmer to design the features.\n",
    "\n",
    "In this way, we will have some ***latent semantic attributes*** that the network can, in principle, learn. BTW, these attributes will probably not be interpretable.\n",
    "\n",
    "In summary, **word embeddings are a representation of the semantics of a word, efficiently encoding semantic information that might be relevant to the task at hand**. We can embed other things too: part of speech tags, parse trees, anything.\n",
    "\n",
    "### Word Embeddings in PyTorch\n",
    "\n",
    "**Similar to how we defined a unique index for each word when making one-hot vectors, we also need to define an index for each word when using embeddings**. These will be keys into a lookup table. In code below, the mapping from words to indices is a dictionary named *word_to_ix*.\n",
    "\n",
    "To index into this table, we must use ***torch.LongTensor***, since the indices are integers, not floats."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x121fa8310>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "torch.manual_seed(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 0.6614,  0.2669,  0.0617,  0.6213, -0.4519],\n",
      "        [-0.1661, -1.5228,  0.3817, -1.0276, -0.5631]],\n",
      "       grad_fn=<EmbeddingBackward>)\n"
     ]
    }
   ],
   "source": [
    "word_to_ix = {'hello': 0, 'world': 1}\n",
    "embeds = nn.Embedding(2, 5)   # V=2, D=5   初始化from标准正态分布\n",
    "lookup_tensor = torch.tensor([0, 1], dtype=torch.long)\n",
    "hello_embed = embeds(lookup_tensor)\n",
    "print(hello_embed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### An Example: N-Gram Language Modeling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(['When', 'forty'], 'winters'), (['forty', 'winters'], 'shall'), (['winters', 'shall'], 'besiege')]\n"
     ]
    }
   ],
   "source": [
    "CONTEXT_SIZE = 2\n",
    "EMBEDDING_DIM = 10\n",
    "test_sentence = \"\"\"When forty winters shall besiege thy brow,\n",
    "And dig deep trenches in thy beauty's field,\n",
    "Thy youth's proud livery so gazed on now,\n",
    "Will be a totter'd weed of small worth held:\n",
    "Then being asked, where all thy beauty lies,\n",
    "Where all the treasure of thy lusty days;\n",
    "To say, within thine own deep sunken eyes,\n",
    "Were an all-eating shame, and thriftless praise.\n",
    "How much more praise deserv'd thy beauty's use,\n",
    "If thou couldst answer 'This fair child of mine\n",
    "Shall sum my count, and make my old excuse,'\n",
    "Proving his beauty by succession thine!\n",
    "This were to be new made when thou art old,\n",
    "And see thy blood warm when thou feel'st it cold.\"\"\".split()\n",
    "\n",
    "# Build a list of tuples: ([word_i-2, word_i-1], target word)   构造出这样的tuple，是trigram的精髓！\n",
    "trigrams = [([test_sentence[i], test_sentence[i + 1]], test_sentence[i + 2]) for i in range(len(test_sentence) - 2)]\n",
    "print(trigrams[:3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = set(test_sentence)\n",
    "word_to_ix = {word: i for i, word in enumerate(vocab)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"feel'st\": 0, 'it': 1, 'to': 2, 'own': 3, \"deserv'd\": 4, 'by': 5, 'blood': 6, 'weed': 7, 'where': 8, 'thriftless': 9, 'his': 10, \"youth's\": 11, 'If': 12, 'proud': 13, \"beauty's\": 14, 'brow,': 15, 'succession': 16, 'on': 17, 'being': 18, 'thou': 19, \"'This\": 20, 'Will': 21, 'winters': 22, 'praise.': 23, 'sunken': 24, 'sum': 25, 'This': 26, 'see': 27, 'praise': 28, 'child': 29, 'And': 30, 'say,': 31, 'besiege': 32, 'fair': 33, 'were': 34, \"totter'd\": 35, 'now,': 36, 'thine!': 37, 'mine': 38, 'a': 39, 'asked,': 40, 'worth': 41, 'shall': 42, 'Where': 43, 'more': 44, 'old': 45, 'answer': 46, 'an': 47, 'the': 48, 'when': 49, 'How': 50, \"excuse,'\": 51, 'Thy': 52, 'eyes,': 53, 'my': 54, 'beauty': 55, 'art': 56, 'be': 57, 'lies,': 58, 'thine': 59, 'forty': 60, 'gazed': 61, 'When': 62, 'field,': 63, 'so': 64, 'and': 65, 'all': 66, 'make': 67, 'To': 68, 'Were': 69, 'much': 70, 'all-eating': 71, 'shame,': 72, 'in': 73, 'treasure': 74, 'cold.': 75, 'new': 76, 'dig': 77, 'days;': 78, 'of': 79, 'Then': 80, 'Shall': 81, 'thy': 82, 'count,': 83, 'old,': 84, 'trenches': 85, 'within': 86, 'small': 87, 'made': 88, 'deep': 89, 'couldst': 90, 'Proving': 91, 'held:': 92, 'warm': 93, 'lusty': 94, 'use,': 95, 'livery': 96}\n"
     ]
    }
   ],
   "source": [
    "print(word_to_ix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "class NGramLanguageModeler(nn.Module):\n",
    "    \n",
    "    def __init__(self, vocab_size, embedding_dim, context_size):\n",
    "        super(NGramLanguageModeler, self).__init__()\n",
    "        self.embeddings = nn.Embedding(vocab_size, embedding_dim)\n",
    "        self.linear1 = nn.Linear(context_size * embedding_dim, 128)  # context_size个word的embedding拼接在一起\n",
    "        self.linear2 = nn.Linear(128, vocab_size)  # 为什么是vocab_size? 因为对于Language Model来说，每次都要输出所有word的概率！\n",
    "        \n",
    "    # Structure: Inputs -> Embedding -> Reshape(view) -> Linear -> ReLU -> Linear -> LogSoftmax(Softmax->Log): 输出的不是probability，而是取了对数(以方便计算对数似然Loss)\n",
    "    def forward(self, inputs):                # inputs: tensor([60, 22])  (2)\n",
    "        embeds = self.embeddings(inputs)      # tensor([[-4.2780e-01, ..., -1.1255e+00], [1.6604e+00, ..., 7.7717e-01]], grad_fn=<EmbeddingBackward>)  (2, 10)\n",
    "        # 所有(context_size个)embedding经reshape后拼接在一起，变成一行\n",
    "        embeds = embeds.view((1, -1))         # tensor([[-4.2780e-01, ..., 7.7717e-01], grad_fn=<ViewBackward>)   (1, 20=context_size*embedding_dim)  \n",
    "        out = F.relu(self.linear1(embeds))    # tensor([[0.4589, ..., 0.3802]], grad_fn=<ReluBackward0>)   (1, 128)   \n",
    "        out = self.linear2(out)               # tensor([[-4.8309, ..., -4.3599]], grad_fn=<AddmmBackward>)  (1, 97=vocab_size)         \n",
    "        log_probs = F.log_softmax(out, dim=1) # tensor([[-4.8075, ..., -4.7506]], grad_fn=<LogSoftmaxBackward>)  (1, 97)   \n",
    "        return log_probs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "losses = []\n",
    "loss_function = nn.NLLLoss()  # Negative Log Likelyhood Loss，一般用于最后一层是LogSoftmax的Network的输出与Label/Target之间计算Loss\n",
    "model = NGramLanguageModeler(len(vocab), EMBEDDING_DIM, CONTEXT_SIZE)\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.001)   # PyTorch中，optimizer在此处与model建立联系！optimizer与model解耦！(Keras中通过model.compile(optimizer=xxx,...))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch:  0\tLoss:  498.0866\n",
      "Epoch:  1\tLoss:  495.7089\n",
      "Epoch:  2\tLoss:  493.3401\n",
      "Epoch:  3\tLoss:  490.9778\n",
      "Epoch:  4\tLoss:  488.6210\n",
      "Epoch:  5\tLoss:  486.2679\n",
      "Epoch:  6\tLoss:  483.9204\n",
      "Epoch:  7\tLoss:  481.5777\n",
      "Epoch:  8\tLoss:  479.2389\n",
      "Epoch:  9\tLoss:  476.9040\n"
     ]
    }
   ],
   "source": [
    "for epoch in range(10):\n",
    "    total_loss = 0\n",
    "    for context, target in trigrams:\n",
    "        # Step1: Recall that torch *accumulates* gradients, so before a new instance, we need to zero out the gradients from old instance\n",
    "        model.zero_grad()\n",
    "        \n",
    "        # Step2: Original Words -> Words Indices -> Tensor Inputs/Target\n",
    "        context_idxs = torch.tensor([word_to_ix[word] for word in context], dtype=torch.long)\n",
    "        target_idx = torch.tensor([word_to_ix[target]], dtype=torch.long)\n",
    "        \n",
    "        # Step3: Forward Pass, getting log probabilities over next words\n",
    "        log_probs = model(context_idxs)\n",
    "        \n",
    "        # Step4: Compute Loss\n",
    "        # 看似是log(probabilities)与target indices之间计算Loss，有些怪怪的，但实际上target indices本质上表示的就是probabilities\n",
    "        # nn.NLLLoss就是用于此种场景，且target indices不需要是one-hot形式，NLLLoss内部会自动处理好\n",
    "        loss = loss_function(log_probs, target_idx)\n",
    "        \n",
    "        # Step5: Backward Pass and update gradients\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        \n",
    "        # Step6: Get loss of this instance, then sum\n",
    "        total_loss += loss.item()\n",
    "        \n",
    "    losses.append(total_loss)\n",
    "    print(f'Epoch: {epoch: 2d}\\tLoss: {total_loss: .4f}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-4.3615, -4.3660, -4.4156, -4.7105, -4.7176, -5.2425, -4.1101, -4.7305,\n",
      "         -4.7247, -4.7705, -4.9730, -5.0012, -4.4570, -4.8580, -4.5421, -5.0889,\n",
      "         -4.7595, -5.1632, -4.5497, -4.1508, -3.6432, -4.4643, -4.3609, -5.2250,\n",
      "         -5.1316, -4.3141, -4.2300, -3.8695, -4.3068, -4.9257, -4.4542, -4.7402,\n",
      "         -4.5531, -4.6898, -4.4954, -4.2568, -4.7212, -4.7477, -4.6154, -4.9766,\n",
      "         -4.6727, -4.6591, -4.5720, -4.6981, -4.6897, -4.6821, -4.9455, -5.0008,\n",
      "         -4.6954, -4.5628, -4.2435, -4.4887, -4.2667, -5.0968, -4.5386, -4.4188,\n",
      "         -4.3432, -4.7809, -4.5174, -4.8471, -5.0712, -5.0107, -4.6593, -4.7827,\n",
      "         -4.9963, -4.1808, -4.8316, -4.5797, -5.1617, -4.4717, -4.6373, -4.6605,\n",
      "         -4.3708, -4.4953, -4.2411, -5.1873, -4.1694, -4.3174, -4.8884, -4.4555,\n",
      "         -4.6635, -4.5252, -4.5162, -4.7773, -3.9616, -4.9337, -4.6259, -4.2613,\n",
      "         -4.7524, -5.0197, -5.0861, -3.9910, -4.4102, -5.0155, -4.6077, -4.5537,\n",
      "         -4.7860]], grad_fn=<LogSoftmaxBackward>)\n",
      "tensor([[0.0128, 0.0127, 0.0121, 0.0090, 0.0089, 0.0053, 0.0164, 0.0088, 0.0089,\n",
      "         0.0085, 0.0069, 0.0067, 0.0116, 0.0078, 0.0107, 0.0062, 0.0086, 0.0057,\n",
      "         0.0106, 0.0158, 0.0262, 0.0115, 0.0128, 0.0054, 0.0059, 0.0134, 0.0146,\n",
      "         0.0209, 0.0135, 0.0073, 0.0116, 0.0087, 0.0105, 0.0092, 0.0112, 0.0142,\n",
      "         0.0089, 0.0087, 0.0099, 0.0069, 0.0093, 0.0095, 0.0103, 0.0091, 0.0092,\n",
      "         0.0093, 0.0071, 0.0067, 0.0091, 0.0104, 0.0144, 0.0112, 0.0140, 0.0061,\n",
      "         0.0107, 0.0120, 0.0130, 0.0084, 0.0109, 0.0079, 0.0063, 0.0067, 0.0095,\n",
      "         0.0084, 0.0068, 0.0153, 0.0080, 0.0103, 0.0057, 0.0114, 0.0097, 0.0095,\n",
      "         0.0126, 0.0112, 0.0144, 0.0056, 0.0155, 0.0133, 0.0075, 0.0116, 0.0094,\n",
      "         0.0108, 0.0109, 0.0084, 0.0190, 0.0072, 0.0098, 0.0141, 0.0086, 0.0066,\n",
      "         0.0062, 0.0185, 0.0122, 0.0066, 0.0100, 0.0105, 0.0083]],\n",
      "       grad_fn=<ExpBackward>)\n",
      "[20, 27, 84, 91, 6, 19, 76, 65, 26, 74, 50, 35, 87, 52, 28, 25, 77, 56, 22, 0, 1, 72, 92, 2, 55, 30, 79, 12, 21, 69, 51, 73, 34, 82, 58, 81, 54, 14, 18, 32, 95, 49, 42, 67, 94, 38, 86, 70, 41, 62, 71, 80, 40, 45, 44, 33, 48, 43, 3, 4, 36, 8, 7, 31, 37, 88, 16, 9, 83, 57, 63, 96, 66, 59, 13, 78, 29, 85, 46, 10, 39, 64, 47, 11, 61, 93, 89, 60, 90, 15, 53, 24, 68, 17, 75, 23, 5]\n",
      "tensor([42])\n"
     ]
    }
   ],
   "source": [
    "context, target = ['forty', 'winters'], 'shall'\n",
    "context_idxs = torch.tensor([word_to_ix[word] for word in context], dtype=torch.long)\n",
    "target_idx = torch.tensor([word_to_ix[target]], dtype=torch.long)\n",
    "log_probs = model(context_idxs)\n",
    "probs = torch.exp(log_probs)    # vocab中各个word的概率\n",
    "idxs_pred = [x.item() for x in probs.argsort(descending=True)[0]]  # 概率从大到小的各个word(的index)\n",
    "print(log_probs)\n",
    "print(probs)\n",
    "print(idxs_pred)\n",
    "print(target_idx)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: Computing Word Embeddings: CBOW\n",
    "\n",
    "CBOW tries to minimize:\n",
    "\n",
    "***-log(p(w_i|C)=-log(Softmax(A(sum(q_w))+b)***\n",
    "\n",
    "where q_w is the embedding for word w, and w is in the context of w_i.\n",
    "\n",
    "Implement this model by filling in the class CBOW. Some tips:\n",
    "\n",
    "- Think about which parameters you need to define.\n",
    "\n",
    "- Make sure you know what shape each operation expects. Use *.view()* if you need to reshape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "CONTEXT_SIZE = 2\n",
    "EMBEDDING_DIM = 10\n",
    "raw_text = \"\"\"We are about to study the idea of a computational process.\n",
    "Computational processes are abstract beings that inhabit computers.\n",
    "As they evolve, processes manipulate other abstract things called data.\n",
    "The evolution of a process is directed by a pattern of rules\n",
    "called a program. People create programs to direct processes. In effect,\n",
    "we conjure the spirits of the computer with our spells.\"\"\".split()\n",
    "vocab = set(raw_text)\n",
    "vocab_size = len(vocab)\n",
    "word_to_ix = {word: i for i, word in enumerate(vocab)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(['We', 'are', 'to', 'study'], 'about'), (['are', 'about', 'study', 'the'], 'to'), (['about', 'to', 'the', 'idea'], 'study')]\n"
     ]
    }
   ],
   "source": [
    "# 生成CBOW所需的训练数据，形如(context, target)\n",
    "data = []\n",
    "for i in range(2, len(raw_text) - 2):\n",
    "    context = [raw_text[i - 2], raw_text[i - 1], raw_text[i + 1], raw_text[i + 2]]\n",
    "    target = raw_text[i]\n",
    "    data.append((context, target))\n",
    "print(data[:3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CBOW(nn.Module):\n",
    "    \n",
    "    def __init__(self, vocab_size, embedding_dim, context_size):\n",
    "        super(CBOW, self).__init__()\n",
    "        self.embedding = nn.Embedding(vocab_size, embedding_dim)\n",
    "        self.linear = nn.Linear(embedding_dim, vocab_size)        # 注意不是context_size * embedding_dim，因为各Embeddings之间不是拼接，是sum\n",
    "    \n",
    "    def forward(self, inputs):\n",
    "        embeds = self.embedding(inputs).sum(dim=0, keepdim=True)  # 注意区别：CBOW中是Sum/Average，而NGram中此处是直接拼接！  (1, 10)\n",
    "        out = self.linear(embeds)              # (1, 49)\n",
    "        log_probs = F.log_softmax(out, dim=1)  # (1, 49)\n",
    "        return log_probs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "losses = []\n",
    "loss_function = nn.NLLLoss()\n",
    "model = CBOW(len(vocab), EMBEDDING_DIM, CONTEXT_SIZE)\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch:     0 - Loss:  273.5698\n",
      "Epoch:   100 - Loss:  122.6719\n",
      "Epoch:   200 - Loss:  72.3473\n",
      "Epoch:   300 - Loss:  46.7546\n",
      "Epoch:   400 - Loss:  31.8023\n",
      "Epoch:   500 - Loss:  22.5712\n",
      "Epoch:   600 - Loss:  16.6680\n",
      "Epoch:   700 - Loss:  12.7883\n",
      "Epoch:   800 - Loss:  10.1564\n",
      "Epoch:   900 - Loss:  8.3068\n",
      "Epoch:  1000 - Loss:  6.9612\n",
      "Epoch:  1100 - Loss:  5.9511\n",
      "Epoch:  1200 - Loss:  5.1718\n",
      "Epoch:  1300 - Loss:  4.5560\n",
      "Epoch:  1400 - Loss:  4.0597\n",
      "Epoch:  1500 - Loss:  3.6526\n",
      "Epoch:  1600 - Loss:  3.3136\n",
      "Epoch:  1700 - Loss:  3.0275\n",
      "Epoch:  1800 - Loss:  2.7834\n",
      "Epoch:  1900 - Loss:  2.5729\n",
      "Epoch:  1999 - Loss:  2.3915\n"
     ]
    }
   ],
   "source": [
    "# 训练model，随后model.embedding.weight即为所求的Word Embeddings？\n",
    "for epoch in range(2000):\n",
    "    total_loss = 0\n",
    "    for i, (context, target) in enumerate(data):\n",
    "        # Step1: Zero out gradients\n",
    "        model.zero_grad()\n",
    "        # Step2: inputs and target --> tensor\n",
    "        context_idxs = torch.tensor([word_to_ix[word] for word in context], dtype=torch.long)\n",
    "        target_idx = torch.tensor([word_to_idx[target]], dtype=torch.long)\n",
    "        # Step3: Forward pass\n",
    "        log_probs = model(context_idxs)\n",
    "        # Step4: Calculate loss\n",
    "        loss = loss_function(log_probs, target_idx)\n",
    "        # Step5: Backward pass\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        # Step6: Add loss\n",
    "        total_loss += loss.item()\n",
    "    \n",
    "    losses.append(total_loss)\n",
    "    if epoch % 100 == 0 or epoch == 1999:\n",
    "        print(f'Epoch: {epoch: 5d} - Loss: {total_loss: .4f}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([49, 10])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([[-1.3441,  0.4166, -1.2872, -1.0262,  0.7593, -2.0134, -0.2606,  1.1865,\n",
       "          1.3127, -1.0511],\n",
       "        [-2.3433,  0.6740,  1.5727, -1.2636, -0.4555, -2.2557,  0.0541, -1.7413,\n",
       "         -0.9428,  3.4371],\n",
       "        [ 0.0912,  0.5830, -0.1556,  3.6323,  2.3521, -1.7655,  0.2268,  0.1228,\n",
       "         -1.3954,  0.4455],\n",
       "        [ 0.3178,  0.0178,  2.3579, -1.6264,  0.6246, -0.3790,  0.1140,  0.5196,\n",
       "         -0.8098, -0.2717],\n",
       "        [ 0.2145,  0.4765,  0.9063,  0.1256,  1.0909, -0.4688,  2.6561,  1.0300,\n",
       "          3.0758,  0.0898]], grad_fn=<SliceBackward>)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(model.embedding.weight.size())\n",
    "model.embedding.weight[:5]           # vocab中前5个word对应的Embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sequence Models and LSTM Networks\n",
    "\n",
    "Before are various feed-forward networks. That is, there is no state maintained by the network at all.\n",
    "\n",
    "Sequence models are central to NLP: they are models where there is some sort of **dependence through time between your inputs**. The classical examples of a sequence model are **HMM** for part-of-speech tagginng and **CRF**.\n",
    "\n",
    "A RNN is a network that maintains some kind of state. For example, its output could be used as part of the next input, so that information can propogate along as the network passes over the sequence.\n",
    "\n",
    "### LSTM in PyTorch\n",
    "\n",
    "Note PyTorch's LSTM expects all of its inputs to be **3D tensors**: (seq_len, batch, input_size)\n",
    "\n",
    "- The 1st axis is the sequence itself\n",
    "- The 2nd indexes instances in the mini-batch  ???\n",
    "- The 3rd indexes elements of the input (**三者中只有input_size才是LSTM定义时的parameter**)\n",
    "\n",
    "For the sentence \"The cow jumped\", out input should look like: **[[q_The], [q_cow], [q_jumped]]** (q_xx mean the row vector of word xx).\n",
    "\n",
    "Since we haven't discussed mini-batching, we just ignore that and assume we will always have just 1 dimension on the 2nd axis.\n",
    "\n",
    "In addition, you could go through the sequence one at a time, in which case the 1st axis will have size 1 also. **--> i.view(1, 1, -1)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x120b311b0>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "torch.manual_seed(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "lstm = nn.LSTM(3, 3)  # Input dim is 3, output dim is 3\n",
    "inputs = [torch.randn(1, 3) for _ in range(5)]         # (5, 1, 3)  1st dim: sequence length 5, 2nd dim: mini-batch index 1, 3rd dim: row vector length 3\n",
    "hidden = (torch.randn(1, 1, 3), torch.randn(1, 1, 3))  # Initialize the hidden state  BiLSTM\n",
    "for i in inputs:\n",
    "    out, hidden = lstm(i.view(1, 1, -1), hidden)       # Step through the sequence one element at a time. After each step, hidden contains the hidden state."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 刘尧：i.view(1, 1, -1)，每个Step的输入原本是(5, 1, 3)中的(1, 3)(共5个Step)，**由于LSTM对输入的要求，需要view为(1, 1, 3)**\n",
    "\n",
    "> 刘尧：实际上LSTM可以同时接受一个Sequence中的所有Step，不用Step one element at a time.\n",
    "\n",
    "In values LSTM returned, ***out*** is all the hidden states throughout the sequence, and ***hidden*** is just the most recent hidden state. (So **out[-1] is same with hidden[0]**)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[[-0.0864,  0.4440, -0.0995]],\n",
      "\n",
      "        [[-0.0642,  0.1129, -0.1143]],\n",
      "\n",
      "        [[-0.0474,  0.1594, -0.1114]],\n",
      "\n",
      "        [[-0.0037,  0.0531, -0.0663]],\n",
      "\n",
      "        [[ 0.0181, -0.0710, -0.2219]]], grad_fn=<StackBackward>)\n",
      "(tensor([[[ 0.0181, -0.0710, -0.2219]]], grad_fn=<StackBackward>), tensor([[[ 0.0483, -0.1326, -0.4540]]], grad_fn=<StackBackward>))\n"
     ]
    }
   ],
   "source": [
    "inputs = torch.cat(inputs).view(len(inputs), 1, -1)    # 变成(5, 1, 3)的Tensor\n",
    "hidden = (torch.randn(1, 1, 3), torch.randn(1, 1, 3))  # BiLSTM\n",
    "out, hidden = lstm(inputs, hidden)\n",
    "print(out)\n",
    "print(hidden)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example: An LSTM for Part-of-Speech Tagging\n",
    "\n",
    "We will not use Viterbi or Forward-Backward or anything like that.\n",
    "\n",
    "Let our input sentence be w1,...,wm, where wi belongs to V, our vocab. Our prediction rule for yhati is: **yhati = argmaxj(logSoftmax(Ahi+b))j**\n",
    "\n",
    "Note that this implies immediately that the dimensionality of the target space of A is |T|, where T is our tag set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'The': 0, 'dog': 1, 'ate': 2, 'the': 3, 'apple': 4, 'Everybody': 5, 'read': 6, 'that': 7, 'book': 8}\n"
     ]
    }
   ],
   "source": [
    "def prepare_sequence(seq, to_ix):\n",
    "    return torch.tensor([to_ix[w] for w in seq], dtype=torch.long)\n",
    "\n",
    "training_data = [\n",
    "    ('The dog ate the apple'.split(), ['DET', 'NN', 'V', 'DET', 'NN']), \n",
    "    ('Everybody read that book'.split(), ['NN', 'V', 'DET', 'NN'])\n",
    "]\n",
    "word_to_ix = {}\n",
    "for sent, tags in training_data:\n",
    "    for word in sent:\n",
    "        if word not in word_to_ix:\n",
    "            word_to_ix[word] = len(word_to_ix)\n",
    "print(word_to_ix)\n",
    "tag_to_ix = {'DET': 0, 'NN': 1, 'V': 2}\n",
    "\n",
    "# These will usually be more like 32 or 64 dimensional. We will keep them small to see how the weights change as we train\n",
    "EMBEDDING_DIM = 6\n",
    "HIDDEN_DIM = 6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LSTMTagger(nn.Module):\n",
    "    \n",
    "    def __init__(self, embedding_dim, hidden_dim, vocab_size, tagset_size):\n",
    "        super(LSTMTagger, self).__init__()\n",
    "        self.hidden_dim = hidden_dim\n",
    "        self.embeddings = nn.Embedding(vocab_size, embedding_dim)\n",
    "        self.lstm = nn.LSTM(embedding_dim, hidden_dim)\n",
    "        self.hidden2tag = nn.Linear(hidden_dim, tagset_size)  # The linear layer maps from hidden state space to tag space\n",
    "    \n",
    "    # Structure: inputs -> Embeddings -> LSTM -> Linear -> LogSoftmax -> tag_scores\n",
    "    def forward(self, sentence):\n",
    "        seq_len = len(sentence)\n",
    "        embeds = self.embeddings(sentence)                    # embeds: (5, 6)  注意！5是某次inputs的sequence长度seq_len，不固定，时刻变化着！\n",
    "        out, hidden = self.lstm(embeds.view(seq_len, 1, -1))  # out: (5, 1, 6)  注意，因为LSTM要求输入是3D，所以此处要view一下，后面还要view回来\n",
    "        tags = self.hidden2tag(out.view(seq_len, -1))         # tags: (5, 3)\n",
    "        tag_scores = F.log_softmax(tags, dim=1)\n",
    "        return tag_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = LSTMTagger(EMBEDDING_DIM, HIDDEN_DIM, len(word_to_ix), len(tag_to_ix))\n",
    "loss_func = nn.NLLLoss()\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see what the scores are before training. Note that element i, j of the output is the score for tag j for word i.\n",
    "\n",
    "注意注意！**Here we don't need to train, so the code is wrapped in torch.no_grad()**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.7417, -1.2835, -1.3999],\n",
      "        [-0.7188, -1.3306, -1.3930],\n",
      "        [-0.7189, -1.3411, -1.3818],\n",
      "        [-0.7587, -1.2966, -1.3537]])\n"
     ]
    }
   ],
   "source": [
    "with torch.no_grad():\n",
    "    inputs = prepare_sequence(training_data[1][0], word_to_ix)\n",
    "    tag_scores = model(inputs)\n",
    "    print(tag_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "【疑问】**要是不加torch.no_grad()内？好像也没啥影响啊？影响在哪？？！！**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.7417, -1.2835, -1.3999],\n",
      "        [-0.7188, -1.3306, -1.3930],\n",
      "        [-0.7189, -1.3411, -1.3818],\n",
      "        [-0.7587, -1.2966, -1.3537]], grad_fn=<LogSoftmaxBackward>)\n"
     ]
    }
   ],
   "source": [
    "inputs = prepare_sequence(training_data[1][0], word_to_ix)\n",
    "tag_scores = model(inputs)\n",
    "print(tag_scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "for epoch in range(300):\n",
    "    for sent, tags in training_data:\n",
    "        model.zero_grad()\n",
    "        inputs = prepare_sequence(sent, word_to_ix)\n",
    "        tags = prepare_sequence(tags, tag_to_ix)\n",
    "        log_probs = model(inputs)\n",
    "        loss = loss_func(log_probs, tags)\n",
    "        loss.backward()\n",
    "        optimizer.step()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-6.1456, -0.0126, -4.5685],\n",
      "        [-2.8703, -3.8968, -0.0801],\n",
      "        [-0.0308, -5.6476, -3.6175],\n",
      "        [-5.4038, -0.0293, -3.7137]])\n"
     ]
    }
   ],
   "source": [
    "with torch.no_grad():\n",
    "    inputs = prepare_sequence(training_data[1][0], word_to_ix)\n",
    "    tag_scores = model(inputs)\n",
    "    print(tag_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: Augmenting the LSTM Part-of-Speech Tagger with Character-level Features\n",
    "\n",
    "Let cw be the character-level representation of word w, and xw be the word embedding as before. Then the input is **the concatenation of cw and xw**. So dimension of inputs is cw's dimension + xw's dimension.\n",
    "\n",
    "Hints:\n",
    "\n",
    "- There will be 2 LSTM's in new model. **The 1st one outputs POS tag scores, and the 2nd one outputs a character-level representation of each word**.\n",
    "\n",
    "- To do a sequence model over characters, you will need to **embed characters**. The character embeddings will be the input to the character LSTM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced: Making Dynamic Decisions and the BiLSTM CRF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
